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FOREWORD 


This  work  unit  was  performed  in  support  of  the  Navy  Technical  Information 
Presentation  Program  (NTIPP)  under  the  auspices  of  the  Naval  Ship  Research  and 
Development  Center,  Bethesda,  Maryland.  The  goal  of  this  program  is  to 
develop  a  system  of  procedures  and  equipments  designed  to  support  and  improve 
the  utility,  preparation,  revision,  storage,  distribution,  and  overall 
management  of  technical  data  for  the  mid-1980  time  period.  The  Navy  Personnel 
Research  and  Development  Center  was  tasked  with  investigating  a  problem 
fundamental  to  this  goal — the  assessment  of  comprehensibility  and  usability  of 
graphic  materials  in  technical  manuals.  This  is  a  preliminary  report  of 
findings  in  this  investigation.  It  details  initial  guidelines  pertinent  to 
graphic  comprehensibility  and  also  provides  an  approach  for  exploring  these 
critical  issues  further.  A  wider  variety  of  Navy  ratings  and  a  broader  range 
of  experience  levels  are  required  to  substantiate  the  generality  of  the 
findings.  Data  to  satisfy  these  requirements  are  now  being  collected  and  will 
be  integrated  with  the  results  contained  in  this  report  to  form  a  more 
comprehensive  future  document. 

NAVPERSRANDCEN  conducted  an  extensive  review  of  technical  graphics  to 
select  appropriate  stimulus  materials  for  experimentation.  This  study  was 
conceived  and  designed  on  the  basis  of  that  review.  Assistance  in  study 
design,  collection  and  analysis  of  data,  and  preparation  of  this  report  was 
provided  under  contract  with  EG&G  Washington  Analytical  Services  Center,  Inc., 
Hydrospace-Challenger  Group,  with  Dr.  M.  Mecherikoff  as  the  contractor's 
Project  Director.  Dr.  T.  E.  Curran  of  NAVPERSRANDCEN  was  the  Principal 
Investigator  for  the  project  and  the  Technical  Monitor  for  the  contract. 


J.  J.  CLARKIN 
Commanding  Officer 
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■  '  Ther  general  approacbwas  r .  .  ...  . 

lW  To  identify  a  limited  aetr  of  features  which  caii  be  hypbt 
facilitating  or  inhibiting  the  usability  of  illustrations . 
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hypothesized 
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;  To  conatruct^ technical  illustrations  displaying  variations  of^these 
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features 


3.  To  measure  the  performance  of  Navy  technicians  extracting  selected' 
information  from  the  illustrations.  _ _ _ 

Specifically,  part  location  and  identification  were  the  technician 
behaviors  selected  for  study.  They  are  common  technician  actions,  the 
elements  in  drawings  which  support  them  (callouts  and  zones)  are  easy  to 
identify  and  manipulate  experimentally,  and  user  performance  can  be  defined 
objectively  in  terms  of  search  time.  A  callout  is  any  label  or  information  on 
the  drawing  itself  which  identifies  a  part.  Callouts  usually  consist  of 
nomenclatures,  reference  designators,  numbers  keyed  to  text  or  tables,  or  a 
combination  of  these.  Zones  are  areas  of  a  drawing  identified  by  alphanumeric 
coordinates  in  the  manner  of  a  road  map.  Examples  of  callouts  and  zones  are 
shown  in  the  figures  of  Appendix  A.  ’£&&&& 


Two  drawings,  a  cross-sectional  view  and  an  exploded  view,  were  used  to 
present  the  stimulus  variations  to  the  subjects.  These  variations  were: 

1.  10,  27,  44,  or  62  callouts  per  drawing. 

2.  Callouts  with  nomenclature,  numbers,  or  both. 


task  of  finding  a-  port  given  a.  callout,  staler 


<•  b.  Forif  numbers  in  .random  order,  as  the  number  of  callouts  increased 
10  to  6  2 ‘^search  tine' increased  by  a  factor  of  three  or  four--.  " 

llf?"  c.  Nomenclature  in  the  callout  along  with  the  number  di<Tnot 
interfere  with  the  search  for  a  number.  ’  i’  - 

"I*  2.  For  the  task  of  finding  a  part  given  nomenclature  (tables  were  in  ^ 
order  of  callout  numbers  rather  than  alphabetical  by  nomenclature):  J2L- 
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a.  For  10  callouts,  scanning  nomenclature  callouts  was  more  efficient 
than  using  a  table. 


v. 


>Jnl-  v«  £•) 


*'5^  b.  When  the  number  of  callouts  was  larger  (27  or  greater),  searching 
a. table,  even  when  not  alphabetical,  was  superior  to  scanning  callouts. 


c.  As  the  number  of  callouts  increased  from  10  to  62,  median  search 
time  increased  by  a  factor  of  about  six.  V 

*3.  For  the  task  of  giving  the  nomenclature  of  a  marked  part  (tables  in  _ 
callout  number  order  are  efficient  for  this  task):  'sJRm 


a.  There  was  a  small  increase  in  search  time  from  10  to  27  callouts 
and  no  increase  beyond  that.  .  .  aflBB 


b.  Where  nomenclature  was  in  the  callouts,  there  were  no  differences 
due  to  increasing  the  number  of  callouts. 


.  For  .lOfcalTouts.'^using:  a.  table  wasias  efficient; .as!  havinggggzp 
nomenclature  ,111  the.  callouts  ;>< for  larger  numbers  /of  callouts there;  waa^an^ 
advantage  to  having;'. nomenclature  in  the vcallouta, even  when  the. numberjo ~  *f' 
cal  Tout  ®  va»  large';  and.-' the  drawing  appeared  cluttered. 
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-  ^A^jjECi re  1  ingfeal loutr  numbers  'and'  lining  t heatup  for*  eas^scanning^y 
inconsistent  results!  An  uncontrolled  variable'  may  have ; clouded/ th»  effects 
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Conclusions 
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FFor  part^ location  byr nomenclature,  use  nomenclature  callout®®* 
number  of  callouts  is  10  or  less;  otherwise,  use  number  callouts  in* sequence 


keyed  to  an  alphabetical  table. 
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V3.  For  part  identification  (finding  the  nomenclature  when  the ■ location  i® 
known),  use  nomenclature  callouts  even  when  the  number  of  callouts  is  large 
and  the  drawing  looks  cluttered  (there  are  no  data  on  the  upper  limit).  d 

4.  If  the  numbers  are  m  sequence,  devices  to  enhance  discriminability 
and  visual  scanning,  such  as  circling  and  lining  up  the  numbers,  are  probably 

5!  Zones  are  not  useful^ for  locating  parts  when  a  number  callout '  mus t 
also  be  used  for  verification.  ~v||| 

With  regard  to  this  study  as  a  prototype,  the  basic  approach  ofisolating 
information  search  behaviors  and  varying  features  of  drawings  which  influence 
the  search  appears  very  promising.  ^  ,  v£i 1 
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Recommendations 


1.  Initiate  changes  to  Navy  technical  manual  procurement  documents  to 
conform  to  the  conclusions  of  this  study.  • 
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INTRODUCTION 


Problem 


Since  illustations  may  easily  comprise  50  percent  or  more  of  a  technical 
manual,  the  effectiveness  of  graphics  in  communicating  technical  information 
is  critical.  Current  graphic  practices,  set  forth  in  military  specifications, 
standards,  handbooks,  and  other  publications,  are  rarely  based  on  objective 
evidence  that  they  improve  utility  and  comprehensibility.  Requirements  in 
different  documents  sometimes  contradict  each  other.  Methods  based  on  valid 
data  are  not  currently  available  either  for  establishing  requirements  for 
procuring  technical  manuals  or  for  objectively  measuring  the  effectiveness  of 
a  particular  illustration  in  supporting  the  job  performance  of  technical 
personnel. 

Objective 

The  objective  of  this  research  is  to  begin  development  of  empirically 
based  guidelines  and  objective  measurement  techniques  to  increase  the 
usability  of  illustrations  in  technical  manuals. 

Specifically,  the  objectives  of  this  study  are: 

1.  To  identify  a  limited  set  of  features  which  can  be  hypothesized  as 
facilitating  or  inhibiting  the  usability  of  illustrations. 

2.  To  construct  technical  illustrations  displaying  variations  of  these 
features . 

3.  To  measure  the  performance  of  Navy  technicians  extracting  selected 
information  from  the  illustrations. 

By  systematically  varying  a  small  number  of  well-defined  features  and 
objectively  measuring  the  effect  on  specific  tasks  simulating  actual  use,  the 
following  goals  can  be  achieved: 

1.  A  quantitative  index  of  effectiveness,  such  as  time-to-locate- 
information,  can  be  associated  with  variations  of  specific  features,  as  a  step 
towards  a  more  generalized  graphic  comprehensibility  metric  in  the  future. 

2.  Requirements  in  specifications  and  guides  can  be  made  less  arbitrary 
by  basing  them  on  performance  data. 

Background 

This  section  describes  the  current  status  in  the  quantification  of  graphic 
comprehensibility,  lists  a  number  of  assumptions  or  principles  which  guided 
the  formulation  of  the  research  approach,  and  presents  the  rationale  for  the 
specific  focus  of  the  present  study. 
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Current  Status 


In  a  recent  comprehensive  survey  of  technical  manual  comprehensibility, 
Curran^  concluded  that  .  .  little  is  known  of  the  variables  making  up 
illustrations  of  various  kinds  and  their  relationship  to  the  comprehensibility 
of  that  illustration.  The  guidance  which  is  offered  technical  illustrators  is 
for  the  most  part  intuitive;  virtually  no  empirical  evidence  is  available." 

There  have,  of  course,  been  serious  efforts  to  control  characteristics  of 
artwork  in  technical  manuals  specifically  for  the  purpose  of  facilitating  the 
transfer  of  information  to  the  user.  At  least  two  major  approaches  can  be 
cited. 

The  first  approach  is  to  implement  an  entire  philosophy  of  how  technical 
information  is  best  presented.  This  is  done  through  a  set  of  procurement 
documents  detailing  the  characteristics  of  the  manual  and  often  the  processes 
by  which  the  product  is  to  be  achieved.  Examples  are  the  Functionally 
Oriented  Maintenance  Manuals  (FOMM),  Fully  Proceduralized  Job  Performance  Aids 
(FPJPA),  NAVAIR's  Work  Package  concept,  and  the  Army's  Integrated  Technical 
Documentation  and  Training  (ITDT)  program.  Artwork  requirements,  often  quite 
detailed,  are  developed  on  the  basis  of  general  experience,  opinion,  aesthetic 
considerations,  past  practice,  reactions  against  past  practice,  and  reasoning 
from  the  overall  philosophy.  Once  the  requirements  are  set  forth  in 
procurement  documents,  technical  manuals  are  procured  in  conformance  with 
them,  and  it  is  nearly  impossible  to  investigate  the  impact  on  user 
performance  of  systematically  varying  the  more  arbitrary  aspects  of  the 
requirements . 

The  second  approach  is  represented  by  several  recent  studies  aiming  to 
relate  personnel  characteristics  such  as  test  scores,  rating,  reading  ability, 
and  experience  level,  together  with  characteristics  of  the  task  and  the  work 
environment,  to  optimum  data  presentation  modes  and  formats.  Recent  work  in 
this  vein  sponsored  by  the  Navy  Technical  Information  Presentation  Program 
(NTIPP)  provides  the  beginnings  of  a  model  by  which  the  best  type  of  graphic 
presentation  can  be  selected  for  a  given  set  of  personnel,  task,  and 
environmental  factors.  This  approach  thus  far  has  been  solely  concerned  with 
choice  of  overall  type  of  format,  and  having  recommended,  for  example,  an 
exploded  view  of  an  assembly,  makes  no  recommendations  about  features  of  the 
drawing  itself  which  may  make  it  easy  or  difficult  to  use. 

These  are  both  positive  and  important  approaches,  in  that  they  focus  on 
tailoring  technical  information  to  match  the  user's  characteristics,  needs, 
and  work  environment.  However,  the.  present  NTIPP-sponsored  study  is  believed 
to  be  the  first  to  employ  the  type  of  detailed  analysis  and  objective 
validation  needed  to  address  the  problem  stated  at  the  outset. 


Curran,  T.  E.  Survey  of  Technical  Manual  Readability  and  Comprehensi¬ 
bility  (Tech.  Rep.  77-37 ) .  San  Diego:  Navy  Personnel  Research  and  Develop- 
ment  Center,  June  1977.  (NTIS  No.  AD-A042  335) 
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A  great  deal  of  technical  artwork  is  controlled  by  MIL-STD-100 
(Engineering  Drawing  Practices).  For  economic  reasons,  it  is  encouraged  that 
engineering  drawings  be  used  or  adapted  for  use  in  technical  manuals,  since 
removing  extraneous  material  from  existing  drawings  is  less  costly  than 
producing  new  artwork  expressly  for  the  maintainer  or  operator.  Because  of  a 
lack  of  the  type  of  research  represented  by  the  present  study,  it  is  not  clear 
that  drawings  designed  to  meet  the  needs  of  designers  and  builders  are  optimal 
for  maintainers  and  operators,  or  to  what  degree  and  in  what  ways  they  are 
suboptimal.  Casual  examination  of  engineering  drawings  suggests  several  ways 
in  which  such  drawings  might  be  improved  for  maintainer  or  operator  use. 
Without  empirical  data  on  the  effects  of  using  current  engineering  drawing 
practices  to  produce  artwork  to  support  Navy  technican  performance,  it  is 
impossible  to  estimate  long  run  cost/benefit  ratios  as  an  alternative  to 
immediate  cost  savings  in  artwork  production. 

Guiding  Assumptions 

To  provide  an  orientation  for  the  development  of  specific  studies  of  the 
interaction  between  a  user  and  his  graphically  presented  data,  the  following 
principles  or  assumptions  were  formulated: 

1.  At  a  particular  point  in  his  overall  sequence  of  behavior  as  a 
maintainer  or  operator,  the  user  has  need  of  information  which  is  graphically 
presented.  At  that  point  he  turns  to  the  graphic  presentation,  having  certain 
prior  information  which  is  his  starting  point  to  find  what  he  needs.  Based  on 
what  he  already  knows  (entry  information),  he  engages  in  a  search,  which  may 
be  long  or  short,  for  what  he  needs  (target  information). 

2.  Certain  identifiable  characteristics  of  arrangement,  labeling, 
referencing,  drawing  practices,  and  so  forth  can  influence  the  effectiveness 
of  his  search.  The  particular  combination  of  factors  contributing  to  an 
effective  search  will  depend  on  the  entry  information  and  the  target 
information.  Graphic  presentations  designed  for  one  type  of  search  will  not 
necessarily  be  most  effective  for  another  type. 

3.  In  some  cases,  the  beginning  and  end  points  of  the  information  search 
are  relatively  easy  to  identify.  Early  research  should  focus  on  such  cases, 
both  to  generate  knowledge  about  them  and  to  provide  insights  into  methodology 
for  less  obvious  and  more  complex  user  activities. 


4.  The  user  himself  is  generally  not  aware  of  the  details  of  his  data 
extraction  process  or  of  the  factors  which  optimize  or  degrade  it.  His 
opinion  about  good  and  bad  artwork  can  suggest  clues  for  further 
investigation,  but  only  direct  performance  measurement  should  be  considered 
conclusive.  The  user  may  become  aware  of  the  search  process  if  it  becomes 
unusually  difficult  or  time  consuming,  but  even  then  it  may  not  be  obvious  to 
him  what  would  improve  the  presentation.  The  user's  attention  is  not  focused 
on  the  data  extraction  process  itself,  nor  should  it  be:  an  effective  search 
for  data  will  be  as  short,  automatic,  and  nonintrusive  as  possible. 


5.  It  is  not  always  obvious  to  an  illustrator  what  the  optimal 
combination  of  characteristics  is,  even  when  the  illustrator  is  aware  of  the 
search  chain  from  entry  information  to  target  information,  and  often  he  is  not 
aware  of  it  or  does  not  consider  it. 

6.  Aesthetic  considerations  alone  will  not  guarantee  optimal  usability 
and  may  sometimes  result  in  degrading  usability.  That  is,  it  may  be  necessary 
to  violate  aesthetic  principles  (for  example,  clean  appearance)  in  order  to 
optimize  information  search. 

7.  Comprehensibility  or  usability  of  a  graphic  presentation  is  not  a 
property  of  the  graphic  presentation  per  se,  but  depends  on  what  information 
is  being  sought  from  it  at  a  particular  moment.  It  reflects  a  relationship 
between  the  characteristics  of  the  graphic  presentation  and  the  task  being 
performed  at  that  moment.  Therefore,  an  index  of  usability  or 
comprehensibility  that  does  not  take  the  intended  use  into  account  is 
seriously  deficient. 


Definition  of  a  Specific  Problem  for  Stud 


Two  important  assumptions  are  that  the  use  to  which  an  illustration  is  to 
be  put  is  critical  in  evaluating  its  usefulness,  and  that  the  evaluation  must 
be  made  in  terms  of  measures  of  user  performance.  Because  of  these,  certain 
very  interesting  types  of  graphics  are  almost  automatically  excluded  for  the 
present.  For  example,  complex,  cognitive  tasks  (e.g.,  troubleshooting)  and 
the  illustrations  which  support  them  (e.g.,  schematics  and  block  diagrams) 
were  not  considered  amenable  at  this  time  to  a  fine-grained  objective  study. 

It  is  expected  that  the  type  of  research  represented  by  the  present  study  will 
suggest  ways  of  objectively  studying  these  other  important  graphic  types  and 
the  behaviors  associated  with  them. 

Location  and  identification  of  parts,  however,  were  judged  to  be  highly 
suitable  user  activities  for  the  present  study  for  the  following  reasons: 

1.  They  are  common  activities  among  technical  manual  users. 

2.  They  are  supported  by  a  number  of  different  types  of  pictorial 
drawings,  such  as  isometric  drawings,  cross-sectional  views,  exploded  views, 
circuit  board  drawings,  and  control  panel  drawings. 

3.  Elements  of  the  drawings  which  are  intended  to  support  this  type  of 
search,  such  as  callouts  and  zones,  are  obvious.  Varying  these  elements  for 
experimental  purposes  is  not  difficult. 

4.  Advice  and  requirements  relating  to  these  elements  are  not  always 
consistent,  and  sometimes  appear  to  derive  from  considerations  of  aesthetics 
and  contractor  convenience  rather  than  effectiveness  and  user  convenience. 

5.  Examples  of  what  appear  to  be  violations  of  human  factors 
considerations  and  even  common  sense  are  not  difficult  to  find  in  recently 
published  technical  manuals. 


6.  The  user's  information  search  task  can  be  easily  and  realistically 
simulated  with  experimental  controls. 

Having  limited  the  present  study  to  part  location  and  identification,  the 
scope  of  questions  which  might  be  asked  is  still  very  broad.  These  include:' 

1.  Should  number  callouts  be  in  sequence?  What  price  in  efficiency  is 
paid  if  they  are  not?  Are  there  cases  where  numerical  sequence  is 
unimportant? 

2.  What  is  the  maximum  number  of  callouts  that  should  appear  on  a 
drawing?  Do  number  callouts  and  nomenclature  callouts  differ  in  this  respect? 

3.  What  is  the  tradeoff  between  having  nomenclature  callouts  on  the 
drawing  versus  putting  the  nomenclature  in  a  table  keyed  to  number  callouts? 

4.  When  reference  designators  are  used  as  callouts,  are  they  less 
discriminate  than  nomenclature  or  numbers,  and  therefore  harder  for  a  user  to 
find  quickly  and  accurately? 

5.  Do  graphic  devices,  such  as  circling  the  numbers  or  using  large,  bold 
type,  help  the  user  scan  more  effectively? 

6.  Should  leaders  (the  line  connecting  the  callout  to  the  part)  or  arrows 
be  short  so  that  the  callout  is  close  to  the  part  it  identifies,  or  is 
scanning  aided  if  the  leaders  are  extended  so  the  callouts  are  arranged  in 
straight  lines? 

7.  If  alphanumeric  zones  are  used,  what  size  of  zone  is  most  effective? 

8.  Should  the  zone  reference  identify  the  location  of  the  center  of  the 
part,  Che  callout,  or  the  arrowhead? 

9.  Zone  designators  on  engineering  drawings  use  the  lower  right  corner  as 
the  origin  and  run  backwards  from  the  normal  reading  direction  (right  to  left 
and  bottom  to  top);  does  this  degrade  search  performance? 

There  are,  of  course,  additional  questions  concerning  the  interaction  of 
the  various  factors  with  each  other  and  with  the  type  of  search  being 
performed.  Some  of  the  above  questions  were  addressed  in  part  in  the  present 
study.  Because  of  the  need  to  limit  the  scope  of  the  present  study,  reference 
designators  were  not  studied,  and  zones  were  represented  only  minimally  to 
collect  information  for  the  design  of  a  future  study. 

Three  common  types  of  information  search  related  to  part  location  and 
identification  were  simulated  in  the  study: 

1.  A  part  is  cited  by  callout  number  in  a  procedure,  explanation,  or 
description:  find  the  part  in  the  drawing. 
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2.  A  part  is  cited  by  nomenclature  in  a  procedure,  explanation,  or 

description:  find  the  part  in  the  drawing. 

3.  A  part  location  in  a  drawing  is  known  (for  instance,  by  recognizing 
its  physical  appearance):  find  its  nomenclature. 


METHOD 


The  general  approach  was  to  identify  tasks  which  closely  simulate  what  a 
technician  does  when  he  is  using  a  graphic  presentation  to  get  needed 
information.  In  this  study  the  tasks  were  restricted  to  location  and 
identification  of  parts,  using  callouts  with  and  without  parts  lists,  and,  to 
a  minor  extent,  using  zones.  As  described  below,  variations  in  number, 
content,  and  arrangement  of  callouts  were  incorporated  into  drawings,  were 
presented  to  subjects  using  five  different  types  of  task  instructions,  and 
were  evaluated  according  to  the  time  required  for  the  specified  information 
search. 


Illustration  Types 

Stimulus  variations  were  incorporated  into  two  basic  drawings  taken  from 
Navy  technical  manuals:  a  cross-sectional  (C/S)  view  of  an  electric  motor, 
and  an  exploded  view  (E/V)  of  a  chart  drive.  These  drawings  are  typical  of 
drawings  found  in  virtually  all  types  of  Navy  technical  manuals.  The 
originals  of  these  two  drawings  were  modified  to  produce  the  experimental 
variations,  examples  of  which  are  reproduced  in  Appendix  A.  All  the  types  of 
variation  are  illustrated  in  these  examples.  The  original  drawings  appear  as 
Figures  A-2  and  A-6,  except  that  A-2  was  hand  lettered  in  its  original  form. 

Variations  of  the  cross-sectional  view  are  not  directly  comparable  to 
those  of  the  exploded  view.  In  particular,  the  cross-sectional  view  as 
originally  drawn  had  nomenclature  callouts,  which  were  retained  in  some  of  the 
variations.  Because  of  the  placement  of  callouts  on  the  exploded  view, 
nomenclature  callouts  were  completely  impractical. 

Number  of  Callouts 

The  number  of  callouts  on  a  drawing  was  varied  in  four  steps:  10,  27,  44, 
and  62.  This  variable  was  applied  in  identical  fashion  to  both  the  cross- 
sectional  view  and  the  exploded  view. 

Content  and  Arrangement  of  Callouts 

There  were  13  variations  in  the  content  and  arrangement  of  callouts.  Five 
of  these  were  applied  to  the  cross-sectional  view: 

1.  Nomenclature  only  (NOMEN). 

2.  Numbers  in  sequence  (NUM-SEQ). 

3.  Numbers  in  random  order  (NUM-RAN). 

4.  Nomenclature  with  numbers  in  sequence  (N/N-SEQ). 


5.  Nomenclature  with  numbers  in  random  order  (N/N-RAN). 
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Table  1 


Description  of  Subject  Tasks 


Task 

Instruction 

Typical  Simulated  Situation 

1 

"Point  to  the  part  with 
callout  number  X. 

Callout  number  appears  in  a  procedure  or 
equipment  description  referencing  a 
figure. 

2 

"Point  to  the  part 
called  Y." 

Nomenclature  appears  in  a  procedure  or 
description  referencing  a  figure. 

3 

"Tell  me  the  nomenclature 
of  the  part  marked  in 
red." 

Physical  appearance  of  the  part  is  known, 
and  the  nomenclature  is  sought. 

4 

"Use  the  zone  system  to 
point  to  the  part  with 
callout  nun&er  X." 

Same  as  task  1. 

5 

"Use  the  zone  system  to 
point  to  the  part  called 
Y." 

Same  as  task  2. 

with  callout  number  28.  Go."  The  subject  would  then  turn  to  the  drawing  and 
find  the  required  part. 

Selection  of  Target  Information 

For  each  test  item,  a  part  was  selected,  the  location  or  identity  of  which 
was  the  target  of  the  information  search.  Targets  were  selected  with  three 
primary  criteria: 

1.  Responding  to  a  test  item  should  not  aid  the  subject  on  a  later  item. 

2.  Targets  were  selected  equally  from  all  areas  of  the  drawings. 

3.  Items  differing  only  by  number  of  callouts  were  assigned  targets  which 
were  not  identical  but  which,  to  the  extent  possible,  were  in  the  same  area  of 
the  drawing. 
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Table  2 


Summary  of  Independent  Variable  Definitions 


Subject  task 


Definition 


Point  to  part  given  callout  number 
Point  to  part  given  nomenclature 
Tell  nomenclature  of  marked  part 

Use  zone  system  to  point  to  part  given  callout  number 
Use  zone  system  to  point  to  part  given  nomenclature 


Number  of  callouts 


Content  and 
arrangement 
variation 


Type  of 
drawing 


Definition 


NUM-SEQ 
NUM-RAN 
N/N-SEQ 
N/N-RAN 
SEQ 
SEQ/C/NE 
SEQ/NC/E 
SEQ/NC/NE 
RAN/ C/E 
RAN/C/NE 
RAN/NC/E 
RAN/NC/NE 


Nomenclature  only 

Numbers  only  in  sequential  order 

Numbers  only  in  random  order 

Nomenclatures  with  numbers  in  sequential  order 
Nomenclatures  with  numbers  in  random  order 
Sequential,  circled,  extended 
Sequential,  circled,  not  extended 
Sequential,  not  circled,  extended 
Sequential,  not  circled,  not  extended 
Random,  circled,  extended 
Random,  circled,  not  extended 
Random,  not  circled,  extended 
Random,  not  circled,  not  extended 


Cross-sectional  view.  Callouts  are  not  extended  to  edge  and  numbers  are 
circled. 

Exploded  view.  Callouts  have  numbers  only.  They  may  be  circled  or  not 
circled,  and  extended  to  edge  or  not  extended. 
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TASK  1 

TASK  2 

TASK  4 

(48  Cells) 

(52  Cells) 

TASK  3 
(52  Cells) 

(4  Cells) 

TASK  5 
(4  Cells) 

Number  of  Callouts 

Number  of  Callouts 

Number  of  Callouts 

CROSS-SECTION 

10  27  44  62 

/ 

10  27  44  62 

10  27  44  62 

NOMEN 

VMammm 

N/N-RAN 

EXPLODED  VIEW 


SEi/C/NE 

SEQ/NC/E 

SEQ/NC/NE 


RAN/C/NE 

RAN/NC/E 

RAN/NC/NE 


Figure  1.  Diagram  of  experimental  design:  stimulus-task  combinations 
used  in  study  are  cross-hatched;  T  indicates  use  of  parts 
list  table  required.  See  Table  2  for  detailed  definitions. 


In  Tasks  2,  3,  4,  and  5,  Che  subject  often  required  information  which  was 
not  part  of  the  callout.  Tables  (parts  lists)  were  used  to  supply  the  missing 
information.  In  Tasks  2  and  3,  the  table  provided  the  bridge  between  callout 
number  and  nomenclature,  and  in  Tasks  4  and  5,  the  table  provided  zone 
information.  All  tables  were  in  the  same  format.  The  number  of  items  in  the 
table  corresponded  Co  the  number  of  callouts  on  the  drawing.  The  items  were 
listed  in  callout  number  sequence.  The  result  was  that  the  tables  were 
efficient  when  entered  by  callout  number,  and  less  efficient  when  entered  by 
nomenclature.  No  task  required  the  subject  to  enter  the  table  by  zone 
designation.  Test  items  which  required  the  use  of  a  table  are  indicated  in 
Figure  1. 

Groups 

A 

Preliminary  trials  using  the  stimulus  materials  indicated  that  to 
administer  all  160  test  items  to  one  subject  would  have  made  the  subject's 
task  excessively  long.  Therefore,  the  160  test  items  were  divided  into  four 
groups  of  40  each,  such  that  the  various  levels  of  each  independent  variable 
were  represented  equally  in  each  group.  One  group  of  40  test  items  was 
administered  to  each  subject. 

Other  Materials 

In  addition  to  the  drawings  for  each  group,  which  were  bound  in  scrambled 
order  for  the  subjects'  use,  there  was  a  corresponding  set  of  item  description 
sheets  to  be  read  to  the  subjects  in  connection  with  each  drawing.  Prior  to 
administration  of  the  test  items,  each  subject  filled  out  a  data  sheet 
soliciting  rate,  educational  level,  Navy  schools  attended,  and  extent  of 
shipboard  experience.  Standard  explanations  and  instructions  were  read  to  the 
subjects.  Since  the  experimenter's  task  was  extremely  tedious,  all  materials 
were  organized  for  easy,  error-free  administration. 

Dependent  Variable 

The  response  variable  was  the  time  in  seconds  (recorded  to  the  nearest 
tenth)  required  to  complete  each  information  search.  After  reading  the  item 
description  sheet,  the  experimenter  said,  "Go,"  whereupon  the  subject  turned 
to  the  next  drawing  and  the  timing  began.  Timing  continued  until  the  subject 
either  pointed  at  a  part  in  the  drawing  or  began  to  speak  his  response,  as 
required.  If  the  response  was  not  correct,  as  happened  occasionally,  the 
experimenter  said,  "That's  not  it,"  and  resumed  the  timing. 


Subjects 

Subjects  for  this  study  were  144  Navy  enlisted  men  in  an  electronic 
rating.  All  subjects  had  training  in  intermediate  electronics  and  93  percent 
had  advanced  (C-school)  training.  Most  (61  percent)  had  shipboard  experience 
maintaining  or  operating  complex  electronic  gear.  Distribution  of  the  sample 
by  rate  is  as  follows: 


Chief  Petty  Officer  1.4% 
Petty  Officer  First  Class  12.8% 
Petty  Officer  Second  Class  33.3% 
Petty  Officer  Third  Class  45.1% 
Seaman  8.3% 


Subjects  were  well-motivated  and  cooperative.  Based  on  spontaneous 
comments  by  subjects,  the  stimulus  variations  were  obvious  enough  that  the 
subjects  could  see  potential  practical  outcomes  of  the  study  related  to  their 
job  tasks. 

Data  Analysis 

Selection  of  an  approach  to  data  analysis  must  take  into  account  the 
following  considerations: 

1.  A  cursory  examination  of  the  data  indicated  that  the  distributions  are 
very  markedly  skewed,  as  would  be  expected  of  time  data  of  this  sort. 

2.  Means  and  variances  of  the  distributions  appear  to  be  correlated  to  a 
substantial  degree. 

3.  Because  the  cells  of  the  design  are  divided  among  four  groups  of 
subjects,  the  observations  may  not  be  statistically  independent. 

4.  It  is  necessary  to  make  numerous  testB  of  significance  on  the  data, 
pooling  observations  from  the  same  subjects  and  from  different  subjects  in 
various  combinations. 

Under  these  circumstances,  the  following  was  regarded  as  the  most 
reasonable  approach: 

1.  Use  appropriate  nonparametric  tests.  There  is  evidence  that  the 
correlations  among  observations  are  negligible;  the  use  of  nonparametric  tests 
appears  justified.  The  Mann-Whitney  U  test  for  the  two-sample  case  and  the 
Kruskal-Wallis  test  for  the  k-sample  case  were  selected  because  they  are  the 
most  powerful  nonparametric  tests  of  their  respective  null-hypotheses. 

2.  Recognize  the  need  to  be  conservative  in  selecting  a  significance 
level.  When  numerous  tests  are  performed,  it  is  expected  that  some  of  the 
"significant"  differences  will  actually  be  due  to  chance.  This  is  less  likely 
if  the  significance  level  is  conservative.  Consider  using  p  <  .001  as  the 
lowest  acceptable  level. 

3.  Recognize  that  the  tests  are  not  strictly  tests  of  central  tendency 
differences  but  "bulk  of  the  distribution"  differences.  That  is,  they  measure 
the  extent  to  which  the  scores  of  each  distribution  exceed  scores  of  the  other 
distribution(s) .  However,  since  the  distributions  are  all  skewed  similarly, 
the  tests  in  most  cases  can  be  reasonably  viewed  as  central  tendency  tests. 
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RESULTS  AND  DISCUSSION 

Teat  Item  Distributions 

Appendix  B  contains  descriptive  data  on  the  distribution  of  scores  for 
each  of  the  160  test  items.  Specifically,  the  tables  in  this  appendix  show 
the  highest  and  lowest  scores,  the  quartiles,  the  range,  the  interquartile 
range,  the  number  of  subjects,  and  the  subject  group  to  which  the  item  was 
administered.  These  distributions  are  displayed  in  groups  of  four.  For  Tasks 
1,  2,  and  3,  the  distributions  for  the  four  number-of-callout  steps  are  shown 
together;  thus  one  can  directly  see  the  effect  of  increasing  the  number  of 
callouts.  For  Tasks  4  and  5,  there  were  only  four  test  items  administered; 
these  are  shown  together  (Tables  B-39  and  B-40).  For  each  set  of  four  test 
items,  the  result  of  the  Kruskal-Wallis  k-sample  test  is  shown.  Numerous 
other  significance  tests  were  performed  and  are  cited  in  the  text  as 
appropriate . 

In  general,  a  large  proportion  of  the  Kruskal-Wallis  and  Mann-Whitney 
tests  performed  indicated  highly  significant  distribution  differences  among 
the  various  conditions.  Since  most  of  the  significant  differences  were  in  the 
expected  direction  and  were  well  beyond  the  .001  level,  it  appears  reasonable 
to  accept  this  as  evidence  that  the  factors  which  were  varied  in  fact 
influenced  performance  (search  time). 

Cross-Sectional  View 

In  Task  1,  the  instruction  to  the  subject  was  to  point  to  the  part  with  a 
given  callout  number.  When  the  numbers  are  in  sequence,  there  is  generally 
only  a  small  increase  in  the  search  time  as  the  number  of  callouts  increases. 
No  consistent  difference  emerges  attributable  to  the  potentially  confusing 
presence  of  nomenclature  on  the  drawing  (tests  of  significance  gave  mixed 
results).  When  the  callout  numbers  are  in  random  order,  there  again  seems  to 
be  little  difference  due  to  the  presence  of  nomenclatures.  However,  the 
search  time  does  tend  to  increase  as  the  number  of  callouts  increases.  One 
could  say  with  reasonable  safety  that  it  takes  approximately  three  to  four 
times  as  long  to  find  a  part  on  the  drawing  when  there  are  62  callouts  as  when 
there  are  10  callouts. 

In  Task  2,  in  which  the  subject  was  to  point  to  the  part  given  the  name  of 
the  part,  nomenclature  in  callouts  is  being  compared  with  the  same 
nomenclature  listed  in  an  accompanying  table.  It  will  be  recalled  that  the 
ordering  of  the  nomenclatures  in  the  table  is  by  callout  number,  so  that  the 
search  for  a  given  part  name,  both  on  the  drawing  and  in  the  table,  was  not 
assisted  by  alphabetical  order  or  any  other  systematic  feature  of  the  part 
name.  The  results  suggest  the  following  interpretations: 

1.  In  the  10-callout  conditions,  the  scanning  of  the  callouts  is 
sufficiently  simple  that  it  is  superior  to  the  use  of  the  table  and  subsequent 
linking  by  callout  number;  the  difference  between  random  and  sequential  order 
in  this  case  is  negligible. 
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2.  As  the  number  of  callouts  increases,  the  search  time  also  increases. 

For  the  conditions  using  nomenclature  in  the  callouts,  the  median  search  time 
increases  by  almost  a  factor  of  six  between  10  callouts  and  62  callouts. 

3.  For  the  conditions  in  which  the  use  of  a  table  is  required,  the 
analysis  is  slightly  more  complex.  The  subject  must  first  locate  the  name  in 
a  table  and  note  the  callout  number,  and  then  use  the  callout  number  to  locate 
the  part  in  the  drawing.  When  the  callout  numbers  are  in  sequence,  most  of 
the  increase  in  search  time  can  be  attributed  to  searching  the  table.  When 
the  callout  numbers  are  in  random  order,  the  time  spent  locating  the  callout 
becomes  significant.  Therefore,  although  both  of  these  conditions  show  an 
increase  in  search  time  with  an  increased  number  of  callouts,  the  condition 
with  random  callout  numbers  shows  a  greater  increase  in  search  time,  as  would..,, 
be  expected.  Considering  the  case  of  62  callouts,  putting  the  callouts  in 
numerical  sequence  cuts  the  search  time  nearly  in  half.  Still  considering  the_4 
case  of  62  callouts,  it  may  be  noted  that  neither  condition  in  which  the 
nomenclature  was  in  the  callouts  was  superior  to  the  condition  in  which  a 
table  was  used  and  the  callout  numbers  were  in  sequence.  This  suggests  that 
when  the  number  of  callouts  is  large,  searching  a  table,  even  when  the  table 
is  not  in  alphabetical  order,  is  more  efficient  than  searching  through  the 
callouts.  This  is  the  reverse  of  the  situation  when  the  number  of  callouts  is 
small.  Using  alphabetical  tables  might  bring  the  search  time  down  as  low  as 
5  seconds  even  for  drawings  with  a  large  number  of  callouts. 

Task  3  responses  are  those  in  which  the  subject  was  to  tell  the  name  of  a 
part  marked  in  red.  Again,  tables  were  necessary  for  some  conditions.  For 
these,  the  subject  found  the  callout  number  from  the  drawing  and  used  that 
number  to  enter  the  table.  Since  the  tables  were  organized  according  to 
callout  number  sequence,  the  tables  were  efficiently  designed  for  this  type  of 
response,  and  the  times  are  relatively  short.  There  appears  to  be  some 
increase  in  search  time  as  the  number  of  callouts  increases  from  10  to  27,  but 
not  much  difference  for  subsequent  increases  in  number  of  callouts.  For  the 
conditions  in  which  the  nomenclature  was  contained  in  the  callouts,  there  were 
no  differences  clearly  attributable  to  increased  number  of  callouts,  and  no 
differences  were  expected.  In  the  case  of  10  callouts,  there  was  no 
difference  between  having  the  nomenclature  in  the  callouts  or  using  a  separate 
table;  for  larger  numbers  of  callouts,  there  appears  to  be  an  advantage  to 
having  the  nomenclature  in  the  callouts,  even  when  the  number  of  callouts  is 
large . 

Exploded  View 

The  results  with  respect  to  random  versus  sequential  order  of  callouts 
were  very  conclusive,  while  circling,  extending  leaders,  and  using  zones 
produced  mixed,  inconclusive  results. 

Random  (RAM)  and  Sequential  (SEQ)  Order 

\j  As  expected,  the  order  of  the  number  callouts  was  the  most  powerful 

vt  variable  manipulated  in  this  study.  For  Tasks  1,  2,  and  3,  within  each 

number-of-cal louts  step,  the  SEQ  conditions  were  combined  and  contrasted  with 
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the  RAN  conditions.  The  results  are  shown  in  Table  3  and  in  Figure  2.  In 
Task  1,  the  numbers  are  used  to  locate  parts  without  a  table.  In  Task  2,  hey 
are  used  with  a  table,  which  accounts  for  the  increase  in  search  time.  When 
the  time  required  to  search  the  table  is  subtracted  out  of  these  data,  the 
Task  2  SEQ  curve  collapses  to  approximate  the  Task  1  SEQ  curve,  in  which  no 
table  is  used.  In  all  four  number-of-callout  steps  in  Tasks  i  and  2,  the 
difference  between  the  RAN  and  SEQ  conditions  is  highly  significant.  The 
magnitude  of  the  difference  is  small  for  10  and  27  callouts,  but  with  a  large 
number  of  callouts,  putting  the  numbers  in  sequence  can  reduce  the  search  tir=p 
by  a  factor  of  three. 
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Table  3 

Medians  cf  Sequence  versus  Random  Conditions 
for  Exploded  View  for  Tasks  1,  2,  and  3 


Conditions 

Humber  of 

Callouts 

10 

27 

44 

62 

Task  1 

se4 

2.00 

2.50 

2.15 

2.10 

RAN 

2.55 

3.55 

6.80 

7.65 

P 

.001 

.001 

.001 

.001 

Task  2 

SEQ 

4.10 

7.00 

6.25 

11.10 

RAN 

5.00 

9.10 

17.60 

17.50 

P 

.001 

.001 

.001 

.001 

Task  3 

SEQ 

2.80 

3.20 

3.20 

4.00 

RAN 

2.40 

3.05 

3.35 

3.35 

P 

.01 

.01 

.01 

.01 

Note.  The  p  values  indicated  were  obtained  by  comparing  the  SEQ  distribution 
with  the  RAN  distribution  using  the  Mann-Whitney  U  test. 
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X  TASK  )  RANDOM 


.  TASK  2  SEQUENCE 

t 


-X  TASK  2  RANDOM 


TASK  3  SEQUENCE 
-X  TASK  3  RANDOM 


TASK  1  SEQUENCE 


Task  3  did  not  require  a  search  for  numbers  on  the  drawings,  and  the 
tables  were  arranged  in  callout  number  sequence.  Therefore,  significant 
differences  were  not  expected,  since  the  order  of  the  numbers  made  no 
difference  in  performing  the  task.  For  44  and  62  callouts,  the  differences 
were  not  significant  even  at  the  .05  level,  although  the  number  of 
observations  was  large.  For  10  and  27  callouts,  the  differences  were 
moderately  significant,  but  they  favored  the  random  arrangement.  The  reason 
for  this  result  is  unknown. 

The  conclusion  to  be  reached  is  that  there  is  a  performance  advantage  to 
using  numbers  in  sequence  rather  than  in  random  order  for  tasks  requiring  the 
user  to  search  for  a  number.  The  advantage  is  small  when  the  number  of 
callouts  is  small,  but  becomes  appreciable  for  drawings  with  large  numbers  of 
callouts . 

Circling  Callouts  and  Extending  Leaders 

Circling  number  callouts  and  extending  leaders  to  enhance  visual  scanning 
were  expected  to  be  much  weaker  in  their  effects  than  the  sequence-random 
variable.  In  fact,  it  was  anticipated  that,  if  the  numbers  were  in  sequence, 
hardly  anything  else  would  matter.  Therefore,  the  C  and  E  variables  would 
prove  beneficial  only  when  the  numbers  were  in  random  order  and  when  there 
were  many  callouts  to  scan.  The  random  conditions  for  44  and  62  callouts. 
Tasks  1  and  2,  were  examined,  and  the  results  were  mixed  and  inconsistent. 

There  are  thought  to  be  two  major  causes  for  these  unsystematic  findings: 

1.  the  position  of  the  target  information  in  the  table  (for  Task  2  only), 
with  resulting  differential  table  search  times,  and 

2.  a  systematic  perceptual  phenomenon  (for  example,  a  figure-ground 
effect  or  a  generalized  scanning  strategy)  which  so  far  is  unexplained. 

On  the  basis  of  a  small  study  of  table  search  times,  it  appears  that  the 
first  of  these  factors  accounts  almost  entirely  for  the  increasing  slope  of 
the  Task  2  SEQ  curve  in  Figure  2.  As  the  number  of  callouts  increases  from  10 
to  62,  more  time  on  the  average  is  required  to  search  the  longer  tables.  This 
factor  also  operates  in  the  Task  2  RAN  condition,  but  seems  much  less  powerful 
than  the  randomness  itself. 

The  second  factor,  on  the  other  hand,  contributes  much  more  variance  to 
the  random  conditions  than  to  the  sequence  conditions.  Its  effect  is  to  make 
the  perception  of  information  on  the  drawing  more  or  less  difficult  depending 
upon  where  the  information  is  located.  Scanning  numbers  in  sequence  is  only 
minimally  affected  by  the  phenomenon,  if  at  all,  but  finding  a  specific  number 
in  a  random  grouping  is  seriously  affected. 


The  effects  of  these  two  factors  were  particularly  evident  in  the 
comparisons  among  the  four  circling  and  extending  conditions.  For  instance, 
in  Table  B-18,  the  b.'-callout  scores  seem  excessively  high,  and  in  Table  B-20, 
the  62-callout  scores  appear  to  be  overly  low.  The  bulk  of  the  inconsistency 


in  each  case  is  explainable  by  the  position  of  the  target  item  in  the  tables. 
In  Table  B-24,  representing  a  random  condition,  the  44-callout  scores  were 
considerably  higher  than  those  for  62  callouts.  In  this  case,  examination  of 
the  stimulus  materials  suggests  that  target  location  on  the  drawing  caused  the 
reversal. 

In  summary,  while  comparisons  among  the  circle/extend  conditions  cannot  be 
made  with  confidence,  the  overall  data  indicate  that  circling  callout  numbers 
and  extending  leaders  are  at  best  weak  variables,  even  when  the  number  of 
callouts  is  large. 

Zones 

The  tasks  performed  by  the  subjects  in  this  study  did  not  lend  themselves 
to  the  use  of  zones.  The  subjects  frequently  seemed  confused  on  these  test 
items  and  the  results  are  not  enlightening.  Comparing  the  Task  4  and  5  items 
with  their  corresponding  Task  1  and  2  items,  the  items  with  zones  invariably 
had  longer  search  times.  Zones  are  not  useful  for  locating  parts  when  a 
number  callout  must  also  be  used  for  verification. 

The  Task  4  distributions  were  not  significantly  different  from  each  other. 
In  Task  5,  the  C/E  cell  was  significantly  lower  than  the  others,  which  would 
be  reasonable  except  for  the  fact  that  the  overall  data  on  circling  and 
extending  leaders  makes  this  finding  suspect. 

Cross-Sectional  versus  Exploded  Views 

The  NUM-SEQ  and  NUM-RAN  conditions  of  the  cross-sectional  drawing  were 
similar  to  the  exploded  view  drawings  in  that  they  had  no  nomenclature.  This 
allowed  a  comparison  between  responses  to  the  two  drawings.  For  each 
number-of-cal louts  step,  a  comparison  was  made  between  NUM-SEQ  and  the 
combined  SEQ  conditions  for  the  exploded  view,  and  between  NUM-RAN  and  the 
combined  RAN  conditions  for  the  exploded  view.  Of  the  24  comparisons,  11  were 
not  significant  at  the  .05  level  and  5  were  significant  at  the  .001  level.  Of 
these  five,  four  indicated  somewhat  longer  times  for  the  cross-sectional  view. 
The  conclusion  is  that  responses  to  the  two  drawings  were  for  the  most  part 
very  similar;  either  type  of  drawing  could  be  used  in  this  type  of  study. 

There  were  some  obviously  confusing  elements  in  this  particular 
cross-sectional  drawing  which  probably  accounted  for  the  significant 
differences . 


Independence  of  Observations 

A  factor  which  could  generate  an  excess  of  significant  sample  differences 
where  no  differences  really  exist  is  the  possibility  of  correlated 
observations.  Such  correlations  might  exist  because  the  same  subjects  were 
used  for  groups  of  test  items.  Thus,  if  speed  of  response  is  a  subject  trait, 
and  one  of  the  groups  of  subjects  happened  to  have  a  large  proportion  of  fast 
subjects,  this  would  be  reflected  in  conditions  scattered  throughout  the 
experiment,  causing  "significant"  differences  not  because  of  the  stimulus 
(cell)  characteristics,  but  because  of  the  group  (subject)  characteristics. 


It  was  impractical  because  of  the  number  of  scores  to  directly  compare  the 
four  group  distributions  with  a  Kruskal-Wallis  test,  and  the  meaning  of  the 
results  would  be  obscure  since  the  effects  of  the  experimental  factors  would 
be  influencing  the  scores  of  the  four  groups  unequally.  Nevertheless,  a 
satisfactory  analysis  is  possible. 

If  search  speed  is  not  a  subject  trait,  but  is  instead  associated  with 
test  items,  then  the  observations  can  be  considered  essentially  independent 
for  the  purpose  of  further  analysis.  Three  analyses  were  performed  using  the 
Kendall  Coefficient  of  Concordance  (W)  method  to  calculate  average  Spearman 
rank  difference  correlations  (rho). 

The  first  analysis  may  be  thought  of  as  pairing  the  36  subjects  in  each 
group  to  yield  all  possible  pairs,  calculating  the  rank  difference  correlation 
(rho)  across  the  40  test  items,  and  finding  the  average  rho  of  the  630  pairs. 
The  average  correlations  turned  out  as  follows: 

Group  Average  Rho 

A  .74 

B  .70 

C  .73 

D  .71 


These  average  correlations  are  significantly  different  from  zero  well  beyond 
the  .001  level. 

The  interpretation  of  such  a  substantial  average  correlation  is  that  the 
subjects  reacted  to  the  items  similarly.  Items  with  short  times  for  one 
subject  tended  to  have  short  times  for  the  others.  There  were  "hard"  items 
and  "easy"  items;  response  time  is  definitely  related  to  the  test  item.  This 
finding  agrees  with  a  cursory  examination  of  the  item  distributions,  but  also 
indicates  the  extent  of  the  relationship.  The  close  agreement  between  groups 
is  one  indication  that  the  groups  were  equivalent. 

The  second  analysis  was  similar,  except  that  the  test  items  were  taken 
pairwise,  with  the  correlations  calculated  across  subjects.  In  this  case,  a 
high  average  correlation  would  mean  that  subjects  are  individually  consistent; 
that  is,  there  are  fast  subjects  and  slow  subjects.  Low  correlations  would 
indicate  that  subject  responses  tended  to  be  independent  of  the  subject;  there 
is  very  little  carryover  from  one  test  item  to  another. 


Group 


Average  Rho 


A 

B 

C 

D 


.16 

.13 

.17 

.14 


These  correlations  are  also  significant  beyond  the  .001  level,  but  are 
uniformly  low,  indicating  almost  no  relationship  between  test  items. 
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Taken  together,  these  results  indicate  that  it  is  justifiable  to  regard 
all  the  scores  as  essentially  independent  measures  reflecting  stimulus 
characteristics . 

A  third  analysis  confirmed  the  second.  Because  of  the  way  the  items  were 
assigned  to  groups,  certain  similar  items  were  administered  to  the  same 
subjects.  In  particular,  there  were  10  pairs  of  Task  1  items  of  this  type,  in 
which  the  two  items  in  the  pair  differed  only  in  having  27  callouts  or  62 
callouts.  The  correlation  was  calculated  for  each  of  the  10  items,  yielding 
rhos  ranging  from  -.08  to  .33.  They  are  uniformly  low,  and  the  highest  fails 
to  meet  the  .05  level  of  significance.  This  is  further  indication  of  the 
independence  of  the  observations. 

Value  of  the  Results 

This  discussion  will  desl  briefly  with  three  questions  which  may  be  raised 
about  this  investigation: 

1.  Aren't  the  results  intuitively  obvious? 

2.  Will  the  results  generalize  to  the  work  environment? 

3.  Even  the  longest  search  times  are  quite  short;  will  these  proposed 
guidelines  really  make  much  difference? 

The  major  results — that  numbers  are  easier  to  find  if  they  are  in 
sequence,  that  tables  should  be  organized  alphabeticaly  or  numerically 
depending  on  the  user's  entry  information,  and  that  drawings  should  be 
different  for  different  uses— seem  rational  and  obvious.  The  importance  of 
this  study  is  twofold:  first,  it  provides  an  objective,  numerical  measure  of 
the  cost  of  violating  these  rational  principles;  second,  the  guidance  and 
requirements  found  in  current  military  procurement  documents  do  not  conform  to 
the  results  of  this  study. 

As  to  the  generalizability  of  the  findings,  it  should  be  noted  that  the 
subjects'  task  in  the  experiment  was  not  truly  a  simulation  of  a  job  task,  but 
rather  a  fractional  part  of  it.  In  important  respects  the  situation  was  quite 
real.  The  search  times  obtained  in  the  study  are  probably  underestimates  of 
on-the-job  times,  however.  The  subjects  were  highly  motivated  to  "beat  the 
clock"  in  spite  of  instructions  to  relax  and  pace  themselves  "normally." 
Ordinary  distractions  and  discomforts  of  the  work  environment  were  absent,  and 
subjects'  whole  attention  was  on  the  graphic  task.  It  is  probable  that  in  the 
work  environment  the  same  relative  magnitudes  or  ratios  would  appear,  but  the 
actual  search  times  would  be  longer. 

Finally,  even  though  the  time  for  each  individual  information  search  is 
short,  these  are  small  tasks  that  occur  with  great  frequency,  so  the  total 
time  could  be  appreciable.  There  is  also  an  annoyance  factor  which  was 
observed  but  not  measured  or  recorded  during  the  data  collection.  Subjects 
who  had  trouble  finding  a  number  or  nomenclature  sometimes  became  quite 
agitated.  On  the  job,  if  the  use  of  a  drawing  appears  to  make  the  job  longer 


instead  of  shorter,  harder  instead  of  easier,  the  technician  may  reject  the 
use  of  the  drawings  or  the  entire  manual  whenever  he  can  get  away  with  it. 
There  is  evidence  that  technicians  do  not  use  a  manual  when  it  appears  harder 
to  use  it  than  not  to.  Every  effort  should  be  made  to  make  the  data  in 
technical  manuals  as  accessible  to  the  user  as  possible. 

Evaluation  of  the  Study 

The  purpose  of  this  study  was  not  only  to  generate  data  on  which  to  base 
guidelines,  but  to  evaluate  the  feasibility  of  attempting  an  objective 
assessment  of  graphic  comprehensibility  or  usability.  As  a  prototype  for 
future  investigations,  the  study  represents  an  approach  which  appears  to  have 
excellent  possibilities  for  solidifying  comprehensibility  requirements  itr 
technical  manual  procurement  documents.  Insofar  as  the  approach  is  new,  it  is 
because  it  focuses  directly  on  the  behavior  of  the  technician  as  he  attempts 
to  extract  needed  data  from  a  drawing.  It  does  not  attempt  a  theoretical 
formulation  of  graphic  comprehensibility,  nor  does  it  attempt  to  operationally 
define  stimulus  features  such  as  "density"  and  "clutter."  It  asks  what  the 
technician  is  doing  with  the  drawing,  and  what  factors  might  be  influencing 
what  he  is  doing.  This  method  has  the  virtue  of  generating  empirical 
relationships  which  should  have  direct  application  and  hopefully  will  also 
provide  a  basis  for  theoretical  developments  regarding  perception  and  human 
information  processing  in  complex  environments. 

The  most  serious  shortcoming  in  this  study  was  the  failure  to  control 
adequately  for  the  two  sources  of  extraneous  variation:  the  location  of  the 

targets  on  the  drawings  and  the  position  of  the  information  in  the 
accompanying  tables.  It  is  apparent  that  the  impact  of  target  location,  both 
on  the  illustration  and  in  the  table,  was  greatly  underestimated.  It  is 
hypothesized  that  systematic  perceptual  variations  caused  some  target  items  to 
take  much  longer  to  locate  than  others  (notably  in  the  random  conditions)  and 
confounded  certain  of  the  results.  Table  search  time,  of  course,  is  part  of 
the  total  performance  time,  but  based  on  data  dealing  with  table  search  time 
only,  this  effect  can  be  dealt  with  mathematically. 

Overall,  the  practical  impact  of  these  factors  is  considered  to  be 
limited.  The  sequence-random  effect  was  very  strong,  and  leads  to  the 
conclusion  that  callouts  should  be  arranged  in  sequence  at  all  times.  The 
hypothesized  perceptual  phenomenon  is  a  problem  only  in  the  random  conditions, 
which  the  results  of  this  study  suggest  should  never  be  used.  It  poses,  at 
this  time,  an  interesting  theoretical  question  but  not  a  practical  one. 
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CONCLUSIONS 


Part  Location  and  Identification  Problem 


1.  For  part  location  by  callout  number,  always  arrange  the  callouts  in 
numerical  sequence  on  the  drawing.  This  is  most  likely  the  best  guideline 
even  when  the  number  of  callouts  is  very  small.  It  is  preferable  to  have  the 
callouts  in  sequence  rather  than  to  have  them  correspond  to  the  numbering  of 
procedural  steps,  if  this  results  in  their  being  out  of  order.  If  the  numbers 
are  in  sequence,  the  number  of  callouts  may  be  quite  large,  certainly  greater 
than  the  62  callouts  used  in  this  study. 

2.  For  part  location  by  nomenclature,  if  the  number  of  callouts  is  10  or 
less,  nomenclature  callouts  may  be  used.  Otherwise,  an  alphabetical  table 
should  be  provided  to  key  the  nomenclatures  to  number  callouts,  which  should 
be  in  sequential  order  on  the  drawing. 

3.  For  part  identification  (finding  the  nomenclature  when  the  location  is 
known),  nomenclature  callouts  are  superior  to  numbers  keyed  to  a  table  even 
when  the  number  of  callouts  is  so  large  that  the  drawing  appears  excessively 
cluttered. 

4.  If  the  numbers  are  in  sequence,  devices  to  enhance  discriminability 
and  visual  scanning  of  number  callouts,  such  as  circling  the  numbers  and 
lining  them  up,  are  probably  unnecessary. 

5.  Zones  are  not  useful  for  locating  parts  when  a  number  callout  must 
also  be  used  for  verification. 

6.  Since  the  guidelines  differ  depending  on  the  type  of  information 
search,  the  drawing  must  be  designed  with  the  information  search  task  in  mind. 

Research  Approach 

The  approach  represented  by  this  study — isolating  the  technical  manual 
users '  information  search  behaviors  and  varying  features  of  drawings  which 
influence  the  speed  and  accuracy  of  his  search — appears  very  promising.  Care 
must  be  taken  in  future  studies  to  randomize,  counterbalance,  or  measure  the 
effect  of  target  location  in  the  stimulus  materials.  Inadequate  control  of 
this  in  the  present  study  caused  confounding  in  certain  desired  comparisons, 
but  need  not  cause  problems  in  future  applications  of  the  approach. 


RECOMMENDATIONS 

1.  Initiate  changes  to  Navy  technical  manual  procurement  documents  to 
conform  to  the  conclusions  of  this  study. 

2.  Pursue  clarification  of  the  graphic  comprehensibility  issue  through 
empirical  studies  of  the  users'  information  search  behavior  and  the  stimulus 
variations  that  influence  its  effectiveness. 
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figure  A-?.  Cross-sectional  view.  Case 
Condition  5  (N/N-RAN). 


Figure  A-3.  Exploded  view.  Case  1 
Condition  6  (SEQ /C/E) 


Figure  A-5.  Exploded  view,  C. 

Condition  6  ( SEQ. 


Table  B-l 


Test  Item  Distribution  Statistics 
Task  1  (Point  to  Part  Given  Callout  Number) 
CROSS-SECTIONAL  VIEW:  NUM-SEQ 


Statistic 

Number 

of  Callouts 

10 

27 

44 

62 

LO  Score 

0.30 

0.30 

0.70 

0.90 

Ql 

1.15 

1.00 

1.25 

1.70 

Q2  (Median) 

1.55 

1.30 

1.80 

2.35 

Q3 

2.00 

2.20 

2.60 

3.05 

HI  Score 

3.60 

3.20 

3.60 

6.50 

HI-LO 

3.30 

2.90 

2.90 

5.60 

Q3-Q1 

0.85 

1.20 

1.35 

1.35 

N 

36 

36 

36 

36 

Group 

A 

D 

C 

D 

Note.  Kruskal-Wallis  k- sample  test  of  distribution  differences  yields  H, 
which  is  distributed  as  Chi-square  with  k-1  degrees  of  freedom. 

H  **  19.98.  Thus,  p  <  .001,  since  p  *  .001  for  H  >  16.27. 
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Table  B-2 


Test  Item  Distribution  Statistics 
Task  1  (Point  to  Part  Given  Callout  Number) 
CROSS-SECTIONAL  VIEW:  NUM-RAN 


Statistic 

Number  of  Callouts 

10 

27 

44 

62 

LO  Score 

0.50 

0.90 

1.10 

1.20 

Ql 

1.25 

3.20 

4.95 

4.00 

Q2  (Median) 

2.00 

3.80 

7.75 

6.50 

Q3 

2.70 

4.80 

15.60 

9.00 

HI  Score 

5.00 

14.50 

40.20 

18.30 

HI-LO 

4.50 

13.60 

39.10 

17.10 

Q3-Q1 

1.45 

1.60 

10.65 

5.00 

N 

36 

36 

36 

36 

Group 

B 

A 

D 

A 

Note.  Kruskal-Wallis  k-sample  test  of  distribution  differences  yields  H, 
which  is  distributed  as  Chi-square  with  k-1  degrees  of  freedom. 

H  =■  69.26.  Thus,  p  <  .001,  since  p  ■  .001  for  H  >  16.27. 


Table  B-3 


Test  Item  Distribution  Statistics 
Task  1  (Point  to  Part  Given  Callout  Number) 
CROSS-SECTIONAL  VIEW:  N/N-SEQ 


Statistic 

Number 

of  Callouts 

10 

27 

44 

62 

LO  Score 

1.00 

0.70 

1.10 

0.60 

Q1  ' 

1.20 

1.65 

2.35 

2.00 

Q2  (Median) 

1.85 

2.80 

3.00 

2.40 

Q3 

2.40 

3.20 

3.70 

3.55 

HI  Score 

5.60 

5.40 

9.20 

9.30 

HI-L0 

4.60 

4.70 

8.10 

8.70 

Q3-Q1 

1.20 

1.55 

1.35 

1.55 

N 

36 

36 

36 

36 

Group 

C 

B 

A 

B 

Note.  Kruskal-Wallis  k-s ample  test  of  distribution  differences  yields  H, 
which  is  distributed  as  Chi-square  with  k-1  degrees  of  freedom. 

H  ■  20.14.  Thus,  p  <  .001,  since  p  ■  .001  for  H  >  16.27. 
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o 


Statistic 


Table  B-4 

Test  Item  Distribution  Statistics 
Task  1  (Point  to  Part  Given  Callout  Number) 
CROSS-SECTIONAL  VIEW:  N/N-RAN 


Number  of  Callouts 


2 


LO 

Score 

0.5 

Q1 

1.2 

Q2 

(Median) 

2.0 

Q3 

HI 

Score 

2.8 

6.4 

>0 

1 

5 

4 

>0 

8 

>5 

10 

Kruskal-Wallis  k-sample  test  of  distribution  differences  yields  H 
which  is  distributed  as  Chi-square  with  k-1  degrees  of  freedom. 

H  *  65.72.  Thus,  p  <  .001,  since  p  -  .001  for  H  >  16.27. 


Table  B-5 


Test  Item  Distribution  Statistics 
Task  1  (Point  to  Part  Given  Callout  Number) 
EXPLODED  VIEW:  SEQ/C/E 


Statistic 

Number 

of  Callouts 

10 

27 

44 

62 

LO  Score 

1.10 

1.10 

0.40 

0.80 

Q1 

1.80 

1.65 

1.10 

1.40 

Q2  (Median) 

2.00 

2.30 

1.25 

2.00 

Q3 

2.65 

3.10 

1.65 

2.75 

HI  Score 

5.00 

6.20 

4.50 

5.60 

HI-LO 

3.90 

5.10 

4.10 

4.80 

Q3-Q1 

0.85 

1.45 

0.55 

1.35 

N 

36 

36 

36 

36 

Group 

A 

D 

C 

D 

Note.  Kruskal-Wallis  k-sample  test  of  distribution  differences  yields  H, 
which  ia  distributed  as  Chi-square  with  k-1  degrees  of  freedom. 

H  ■  31.1.  Thus,  p  <  .001,  since  p  ■  .001  for  H  >  16.27. 
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Table  B-6 

Test  Item  Distribution  Statistics 
Task  1  (Point  to  Part  Given  Callout  Number) 
EXPLODED  VIEW:  SEQ/C/NE 


Number  of  Callouts 


2 


LO  Score 

0 

Q1 

1 

Q2  (Median) 

1 

Q3 

2 

HI  Score 

5 

0.40 

0 

2.00 

1 

3.85 

1 

5.70 

2 

5.00 

4 

Kruskal-Wallis  k-s ample  test  of  distribution  differences  yields  H 
which  is  distributed  as  Chi-square  with  k-1  degrees  of  freedom. 

H  ■  32.2.  Thus,  p  <  .001,  since  p  *  .001  for  H  >  16.27. 


Table  B-7 


Test  Item  Distribution  Statistics 
Task  1  (Point  to  Part  Given  Callout  Number) 
EXPLODED  VIEW:  SEQ/NC/E 


Statistic 

Number 

of  Callouts 

10 

27 

44 

62 

LO  Score 

0.90 

1.60 

1.20 

0.70 

Q1 

1.80 

2.30 

2.25 

1.20 

Q2  (Median) 

2.45 

3.00 

2.80 

1.95 

Q3 

3.10 

4.10 

3.25 

3.50 

HI  Score 

6.90 

8.60 

5.80 

8.00 

HI-LO 

6.00 

7.00 

4.60 

7.30 

Q3-Q1 

1.30 

1.80 

1.00 

2.30 

N 

36 

36 

36 

36 

Group 

C 

B 

A 

B 

Note.  Rruskal-Wallis  k-sample  test  of  distribution  differences  yields  H 
which  is  distributed  as  Chi-square  with  k-1  degrees  of  freedom. 

H  ■  12.87.  Thus,  p  <  .01,  since  p  *  .01  for  H  >  11.34. 


Table  B-8 


Test  Item  Distribution  Statistics 
Task  1  (Point  to  Part  Given  Callout  Number) 
EXPLODED  VIEW:  SEQ/NC/NE 


Statistic 

Number  of 

Callouts 

10 

27 

44 

62 

LO  Score 

0.60 

1.00 

0.90 

0.50 

Q1 

1.30 

2.00 

1.20 

2.60 

Q2  (Median) 

2.20 

2.95 

1.85 

4.65 

Q3 

3.85 

5.20 

2.90 

7.05 

HI  Score 

9.00 

26.20 

7.10 

12.50 

HI-LO 

8.40 

25.20 

6.20 

12.00 

Q3-Q1 

2.55 

3.20 

1.70 

4.45 

N 

36 

36 

36 

36 

Group 

D 

C 

B 

A 

Note.  Kruskal-Wallis  k- sample  test  of  distribution  differences  yields  H, 
which  is  distributed  as  Chi-square  with  k-1  degrees  of  freedom. 

H  *  22.96.  Thus,  p  <  .001,  since  p  *  .001  for  H  >  16.27. 
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Table  B-9 


Test  Item  Distribution  Statistics 
Task  1  (Point  to  Part  Given  Callout  Number) 
EXPLODED  VIEW:  RAN/C/E 


Statistic 

Number  of  Callouts 

10 

27 

44 

62 

LO  Score 

0.90 

0.80 

2.30 

1.70 

Q1 

2.00 

2.05 

2.95 

4.65 

Q2  (Median) 

2.20 

3.40 

3.95 

8.55 

Q3 

2.95 

6.25 

5.05 

16.10 

HI  Score 

7.50 

15.80 

13.80 

24.20 

HI-LO 

6.60 

15.00 

11.50 

22.50 

Q3-Q1 

0.95 

4.20 

2.10 

11.45 

N 

36 

36 

36 

36 

Group 

D 

C 

B 

C 

Note.  Kruskal-Wallis  k-sample  test  of  distribution  differences  yields  H, 
which  is  distributed  as  Chi-square  with  k-1  degrees  of  freedom. 

H  ■  48.43.  Thus,  p  <  .001,  since  p  ■  .001  for  H  >  16.27. 
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Table  B-10 

Test  Item  Distribution  Statistics 
Task  1  (Point  to  Part  Given  Callout  Number) 
EXPLODED  VIEW:  RAN/C/NE 


Number  of  Callouts 


2 


LO 

Score 

0 

Q1 

1 

Q2 

(Median) 

2 

Q3 

3 

HI 

Score 

6 

i0 

1.00 

10 

2.35 

5 

3.30 

15 

5.00 

Kruskal-Wallis  k-sample  test  of  distribution  differences  yields  H, 
which  is  distributed  as  Chi-square  with  k-1  degrees  of  freedom. 

H  ■  74.36.  Thus,  p  <  .001,  since  p  "  .001  for  H  >  16.27. 


Table  B-ll 


Test  Item  Distribution  Statistics 
Task  1  (Point  to  Part  Given  Callout  Number) 
EXPLODED  VIEW:  RAN/NC/E 


Statistic 

Number  of  Callouts 

10 

27 

44 

62 

LO  Score 

1.30 

1.50 

2.00 

2.10 

Q1 

2.90 

2.20 

4.10 

4.70 

Q2  (Median) 

4.15 

3.20 

6.95 

6.40 

Q3 

5.55 

4.60 

12.95 

10.95 

HI  Score 

23.60 

11.10 

26.50 

35.20 

HI-LO 

22.30 

9.60 

24.50 

33.10 

Q3-Q1 

2.65 

2.40 

8.85 

6.25 

N 

36 

36 

36 

36 

Group 

B 

A 

D 

A 

Note .  Kruskal-Wallis  k-s ample  test  of  distribution  differences  yields  H, 
which  is  distributed  as  Chi-square  with  k-1  degrees  of  freedom. 

H  ■  31.16.  Thus,  p  <  .001,  since  p  *  .001  for  H  >  16.27. 
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Table  B— 12 


Test  Item  Distribution  Statistics 
Task  1  (Point  to  Part  Given  Callout  Number) 
EXPLODED  VIEW:  RAN/NC/NE 


Statistic 

Number  of  Callouts 

10 

27 

44 

62 

LO  Score 

0.60 

0.80 

1.40 

1.60 

Q1 

1.25 

2.55 

4.80 

3.00 

Q2  (Median) 

1.85 

4.85 

9.20 

6.00 

Q3 

2.80 

8.35 

11.20 

10.70 

HI  Score 

5.00 

46.00 

28.30 

21.20 

HI-LO 

4.40 

45.20 

26.90 

19.60 

Q3-Q1 

1.55 

5.80 

6.40 

7.70 

N 

36 

36 

36 

36 

Group 

A 

D 

C 

D 

Note.  Kruskal-Wallis  k-sample  test  of  distribution  differences  yields  H, 
which  is  distributed  as  Chi-square  with  k-1  degrees  of  freedom. 

H  *  52.18.  Thus,  p  <  .001,  since  p  “  .001  for  H  >  16.27. 
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Statistic 


LO  Score 

Q1 

Q2  (Median) 

Q3 

HI  Score 
HI-LO 


Table  B-13 

Test  Item  Distribution  Statistics 
Task  2  (Point  to  Part  Given  Nomenclature) 
CROSS-SECTIONAL  VIEW:  NOMEN 


Number  of  Callouts 


2 


90 

1.4 

10 

3.9 

55 

6.2 

45 

8.9 

Kruskal-Wallis  k-sample  test  of  distribution  differences  yields  H 
which  is  distributed  as  Chi-square  with  k-1  degrees  of  freedom. 

H  “  59.93.  Thus,  p  <  .001,  since  p  *  .001  for  H  >  16.27. 
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Table  B-14 

Test  Item  Distribution  Statistics 
Task  2  (Point  to  Part  Given  Nomenclature) 
CROSS-SECTIONAL  VIEW:  NUM-SEQ 


Statistic 

Number  of  Callouts 

10 

27 

44 

62 

LO  Score 

2.70 

3.10 

3.80 

4.60 

Q1 

3.50 

6.90 

6.30 

6.70 

Q2  (Median) 

4.65 

8.20 

9.05 

8.70 

Q3 

5.00 

11.45 

10.55 

11.05 

HI  Score 

11.20 

24.30 

21.30 

27.90 

HI-LO 

8.50 

21.20 

17.50 

23.30 

Q3-Q1 

1.50 

4.55 

4.25 

4.35 

N 

36 

36 

36 

36 

Group 

B 

C 

D 

A 

Note .  Kruskal-Wallis  k-saaple  test  of  distribution  differences  yields  H, 
which  is  distributed  as  Chi-square  with  k-1  degrees  of  freedom. 

H  ■  56.94.  Thus,  p  <  .001,  since  p  ■  .001  for  H  >16.27. 


Table  B-15 


Test  Item  Distribution  Statistics 


Task  2  (Point  to  Part  Given  Nomenclature) 
CROSS-SECTIONAL  VIEW:  NUM-RAN 


Statistic 

Number  of  Callouts 

10 

27 

44 

62 

LO  Score 

2.10 

2.90 

5.50 

5.90 

Q1 

3.80 

9.05 

12.20 

12.70 

Q2  (Median) 

4.60 

11.30 

15.65 

16.60 

Q3 

5.10 

13.65 

19.45 

18.20 

HI  Score 

8.30 

26.20 

32.00 

41.20 

HI-LO 

6.20 

23.30 

26.50 

35.30 

Q3-Q1 

1.30 

4.60 

7.25 

5.50 

N 

36 

36 

36 

36 

Group 

C 

D 

A 

B 

Note.  Kruskal-Wallis  k-sample  test  of  distribution  differences  yields  H, 
which  is  distributed  as  Chi-square  with  k-1  degrees  of  freedom. 

H  ■  85.29.  Thus,  p  <  .001,  since  p  ■  .001  for  H  >  16.27. 


Table  B-16 


Test  Item  Distribution  Statistics 
Task  2  (Point  to  Part  Given  Nomenclature) 
CROSS-SECTIONAL  VIEW:  N/N-SEQ 


Statistic 


Number  of  Callouts 


10  27  44  62 


LO  Score 

0.50 

2.30 

1.30 

1.80 

Q1 

5.90 

Q2  (Median) 

2.30 

8.05 

10.25 

13.25 

Q3 

11.90 

15.15 

HI  Score 

4.10 

104.00 

38.00 

46.00 

HI-LO 

3.60 

101.70 

36.70 

44.20 

Q3-Q1 

1.70 

9.15 

N 

36 

36 

36 

36 

Group 

D 

A 

B 

C 

Note.  Kruskal-Wallis  k-sample  test  of  distribution  differences  yields  H 
which  is  distributed  as  Chi-square  with  k-1  degrees  of  freedom. 

H  *  70.92.  Thus,  p  <  .001,  since  p  ■  .001  for  H  >  16.27. 


Table  B-17 


Test  Item  Distribution  Statistics 
Task  2  (Point  to  Part  Given  Nomenclature) 
CROSS-SECTIONAL  VIEW:  N/N-RAN 


Statistic 

Number  of  Callouts 

10 

27 

44 

62 

LO  Score 

1.00 

0.90 

1.50 

2.50 

Q1 

1.45 

3.25 

3.50 

6.85 

Q2  (Median) 

2.05 

4.70 

5.45 

11.75 

Q3 

3.00 

7.65 

8.55 

26.95 

HI  Score 

4.20 

14.30 

21.00 

63.10 

HI-LO 

3.20 

13.40 

19.50 

60.60 

Q3-Q1 

1.55 

4.40 

5.05 

20.10 

N 

36 

36 

36 

36 

Group 

A 

B 

C 

D 

Note.  Kruskal-Wallis  k-sample  test  of  distribution  differences  yields  H, 
which  is  distributed  as  Chi-square  with  k-1  degrees  of  freedom. 

H  ■  78.19.  Thus,  p  <  .001,  since  p  *  .001  for  H  >  16.27. 


Table  B-18 


Test  Item  Distribution  Statistics 
Task  2  (Point  to  Part  Given  Nomenclature) 
EXPLODED  VIEW:  SEQ/C/E 


Statistic 

Number  of  Callouts 

10 

27 

44 

62 

L0  Score 

2.70 

2.90 

2.00 

5.10 

Q1 

3.60 

4.00 

4.20 

8.90 

Q2  (Median) 

4.60 

4.80 

5.30 

10.60 

Q3 

5.10 

5.90 

7.45 

13.25 

HI  Score 

8.90 

13.10 

18.50 

58.00 

HI-LO 

6.20 

10.20 

16.50 

52.90 

Q3-Q1 

1.50 

1.90 

3.25 

4.35 

N 

36 

36 

36 

36 

Group 

B 

A 

D 

C 

Note.  Kruskal-Wallis  k- sample  test  of  distribution  differences  yields  H 
which  is  distributed  as  Chi-square  with  k-1  degrees  of  freedom. 

H  “  66.89.  Thus,  p  <.001,  since  p  ■  .001  for  H  >  16.27. 


Table  B-19 


Test  Item.  Distribution  Statistics 
Task  2  (Point  to  Part  Given  Nomenclature) 
EXPLODED  VIEW:  SEQ/C/NE 


Statistic 

Number  of  Callouts 

10 

27 

44 

62 

L0  Score 

1.30 

4.60 

4.00 

3.60 

Q1 

2.25 

6.10 

6.90 

9.80 

Q2  (Median) 

3.15 

8.20 

8.80 

15.15 

Q3 

4.00 

10.05 

11.65 

17.50 

HI  Score 

5.00 

33.10 

20.20 

52.20 

HI-LO 

3.70 

28.50 

16.20 

48.60 

Q3-Q1 

1.75 

3.95 

4.75 

7.70 

N 

36 

36 

36 

36 

Group 

C 

B 

A 

D 

Note.  Kruskal-Wallis  k-sample  test  of  distribution  differences  yields  H, 
which  is  distributed  as  Chi-square  with  k-1  degrees  of  freedom. 

H  *  86.85.  Thus,  p  <  .001,  since  p  *  .001  for  H  >  16.27. 


Table  B-20 


Test  Item  Distribution  Statistics 
Task  2  (Point  to  Part  Given  Nomenclature) 
EXPLODED  VIEW:  SEQ/NC/E 


Statistic 

Number  of  Callouts 

10 

27 

44 

62 

LO  Score 

1.80 

4.10 

3.60 

2.20 

Q1 

A. 15 

6.10 

5.45 

4.75 

Q2  (Median) 

4.75 

8.00 

10.80 

6.90 

Q3 

6.05 

10.20 

15.20 

9.65 

HI  Score 

9.10 

19.60 

23.00 

23.20 

HI-LO 

7.30 

15.50 

19.40 

21.00 

Q3-Q1 

1.90 

4.10 

9.75 

4.90 

N 

36 

36 

36 

36 

Group 

D 

C 

B 

A 

Note,  Kruskal-Wallia  k-sample  test  of  distribution  differences  yields  H 
which  is  distributed  as  Chi-square  with  k-1  degrees  of  freedom. 

H  -  30.28.  Thus,  p  <  .001,  since  p  *  .001  for  H  >  16.27. 
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Table  B-21 


Test  Item  Distribution  Statistics 


Task  2  (Point  to  Part  Given  Nomenclature) 
EXPLODED  VIEW:  SEQ/NC/NE 


Statistic 

. 

Number  of  Callouts 

27 

44 

62 

LO  Score 

1.70 

3.00 

2.00 

5.20 

Q1 

3.00 

5.25 

2.90 

10.20 

Q2  (Median) 

4.00 

7.65 

3.50 

13.95 

Q3 

5.30 

12.15 

4.70 

21.30 

HI  Score 

19.90  - 

24.80 

18.00 

35.60 

HI-LO 

18.20 

21.80 

16.00 

30.40 

Q3-Q1 

2.30 

6.90 

1.80 

11.10 

N 

36 

36 

36 

36 

Group 

A 

D 

C 

B 

Note.  Kruskal-Wallis  k-a ample  test  of  distribution  differences  yields  H, 
which  is  distributed  as  Chi-square  with  k-1  degrees  of  freedom. 

H  -  80.4S.  Thus,  p  <  .001,  since  p  ■  .001  for  H  >  16.27. 


Note.  Kruskal-Wallis  k-saaple  test  of  distribution  differences  yields  B 
which  is  distributed  as  Chi-square  with  k-1  degrees  of  freedom. 

H  »  84.09.  Thus,  p  <  .001,  since  p  “  .001  for  H  >  16.27. 
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Table  B-23 


Test  Item  Distribution  Statistics 
Task  2  (Point  to  Part  Given  Nomenclature) 
EXPLODED  VIEW:  RAN/C/NE 


Statistic 

Number  of  Callouts 

10 

27 

44 

62 

LO  Score 

1.30 

3.60 

5.90 

4.40 

Q1 

3.65 

7.10 

11.20 

10.30 

Q2  (Median) 

4.90 

8.55 

17.55 

13.40 

Q3 

6.05 

10.25 

23.70 

16.40 

HI  Score 

11.00 

33.30 

64.00 

31.10 

HI-LO 

9.70 

29.70 

58.10 

26.70 

Q3-Q1 

2.40 

3.15 

12.50 

6.10 

N 

36 

36 

36 

36 

Group 

D 

C 

B 

A 

Note.  Kruskal-Wallis  k-sample  test  of  distribution  differences  yields  H, 
which  is  distributed  as  Chi~square  with  k-1  degrees  of  freedom. 

H  ■  82.01.  Thus,  p  <  .001,  since  p  ■  .001  for  H  >  16.27. 


Table  B-24 

Test  Item  Distribution  Statistics 
Task  2  (Point  to  Part  Given  Nomenclature) 
EXPLODED  VIEW:  RAN/NC/E 


Statistic 


LO  Score 

Q1 

Q2  (Median) 

Q3 

HI  Score 


Number  of  Callouts 


2 


r 0 

7 

10 

17 

:5 

27 

5 

34 

0 

82 

Kruskal-Wallis  k-sample  test  of  distribution  differences  yields  H 
which  is  distributed  as  Chi-square  with  k-1  degrees  of  freedom. 

H  =»  83.84.  Thus,  p  <  .001,  since  p  *  .001  for  H  >  16.27. 
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Table  B-25 


Test  Item  Distribution  Statistics 
Task  2  (Point  to  Part  Given  Nomenclature) 


EXPLODED  VIEW:  RAN/NC/NE 


Statistic 

Number  of  Callouts 

10 

27 

44 

62 

LO  Score 

1.30 

3.60 

7.60 

5.20 

Q1 

2.30 

6.35 

12.65 

11.25 

Q2  (Median) 

3.20 

8.30 

15.90 

19.90 

Q3 

4.60 

11.35 

20.60 

33.65 

HI  Score 

8.70 

15.30 

35.20 

59.80 

HI-LO 

7.40 

11.70 

27.60 

54.60 

Q3-Q1 

2.30 

5.00 

7.95 

22.40 

N 

36 

36 

36 

36 

Group 

B 

A 

D 

C 

Note.  Kruskal-Wallis  k-sample  test  of  distribution  differences  yields  H, 
which  is  distributed  as  Chi-square  with  k-l  degrees  of  freedom. 

H  ■  94.78.  Thus ,  p  <  .001,  since  p  ■  .001  for  H  >  16.27. 
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Table  B-26 


Test  Item  Distribution  Statistics 
Task  3  (Tell  Nomenclature  of  Marked  Part) 
CROSS-SECTIONAL  VIEW;  NOMEN 


Statistic 

Number 

of  Callouts 

10 

27 

44 

62 

LO  Score 

0.50 

0.80 

0.80 

1.00 

Q1 

0.80 

1.20 

2.00 

1.70 

Q2  (Median) 

1.20 

1.30 

2.25 

2.00 

Q3 

1.50 

1.95 

3.00 

2.60 

HI  Score 

2.20 

3.60 

8.10 

11.10 

HI-LO 

1.70 

2.80 

7.30 

10.10 

Q3-Q1 

0.70 

0.75 

1.00 

0.90 

N 

36 

36 

36 

36 

Croup 

B 

C 

D 

A 

Note.  Kruskal-Wallis  k-s ample  test  of  distribution  differences  yields  H, 
which  is  distributed  as  Chi-square  with  k-1  degrees  of  freedom. 

H  ■  53.94.  Thus,  p  <  .001,  since  p  ■  .001  for  H  >  16.27. 
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Table  B-27 


Test  Item  Distribution  Statistics 
Task  3  (Tell  Nomenclature  of  Marked  Part) 
CROSS-SECTIONAL  VIEW:  NUM-SEQ 


Statistic 


Number  of  Callouts 


2 


Kruskal-Wallis  k-sample  test  of  distribution  differences  yields  H 
which  is  distributed  as  Chi-square  with  k-1  degrees  of  freedom. 

H  ■  89.34.  Thus,  p  <  .001,  since  p  ■  .001  for  H  >  16.27. 
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Table  B-28 

Test  Item  Distribution  Statistics 
Task  3  (Tell  Nomenclature  of  Harked  Part) 
CROSS-SECTIONAL  VIEW:  NUM-RAN 


K>>3 


Statistic 

Number 

of  Callouts 

10 

27 

44 

62 

LO  Score 

1.30 

2.80 

1.80 

2.70 

Q1 

2.00 

3.80 

2.80 

4.90 

Q2  (Median) 

2.20 

4.60 

3.70 

5.75 

Q3 

2.80 

5.20 

4.45 

7.05 

HI  Score 

3.80 

7.00 

6.20 

10.20 

HI-L0 

2.50 

4.20 

4.40 

7.50 

Q3-Q1 

0.80 

1.40 

1.65 

2.15 

N 

36 

36 

36 

36 

Group 

D 

A 

B 

C 

Note.  Kruskal-Wallis  k-sample  test  of  distribution  differences  yields  H, 
which  is  distributed  as  Chi-square  with  k-1  degrees  of  freedom. 

H  ■  84.07.  Thus,  p  <  .001,  since  p  ■  .001  for  H  >  16.27. 


Table  B-29 


Test  Item  Distribution  Statistics 
Task  3  (Tell  Nomenclature  of  Marked  Part) 
CROSS-SECTIONAL  VIEW:  N/N-SEQ 


Statistic 

Number  of  Callouts 

10 

27 

44 

62 

LO  Score 

1.00 

0.80 

0.50 

1.00 

Q1 

1.50 

1.45 

1.35 

1.70 

Q2  (Median) 

2.00 

1.80 

2.00 

2.05 

Q3 

2.20 

2.60 

2.60 

2.65 

HI  Score 

4.00 

4.80 

11.10 

3.60 

HI-LO 

3.00 

4.00 

10.60 

2.60 

Q3-Q1 

0.70 

1.15 

1.25 

0.95 

N 

36 

36 

36 

36 

Croup 

A 

B 

C 

D 

Note.  Kruskal-Wallis  k-sample  teat  of  distribution  differences  yields  H, 
which  is  distributed  as  Chi-square  with  k-1  degrees  of  freedom. 

H  ■  2.4.  Thus,  p  >  .05,  since  p  ■  .05  for  H  >  7.82. 


B-29 


5 


Statistic 


Table  B-30 

Test  Item  Distribution  Statistics 
Task  3  (Tell  Nomenclature  of  Marked  Part) 
CROSS-SECTIONAL  VIEW:  N/N-RAN 


Number  of  Callouts 


LO  Score 

0 

Q1 

1 

Q2  (Median) 

2 

Q3 

2 

HI  Score 

8 

HI-LO 

7 

0 

1.00 

1 

0 

2.05 

2 

15 

2.75 

3 

0 

4.25 

4 

12.70 

12 

Note_.  Kruskal-Wallis  k-sample  test  of  distribution  differences  yields  H 
which  is  distributed  as  Chi-square  with  k-1  degrees  of  freedom. 

H  *•  37.45.  Thus,  p  <  .001,  since  p  “  .001  for  H  >  16.27. 


Table  B-31 


Test  Item  Distribution  Statistics 
Task  3  (Tell  Nomenclature  of  Harked  Part) 
EXPLODED  VIEW:  SEQ/C/E 


Statistic 

Number  of  Callouts 

10 

27 

44 

62 

LO  Score 

1.40 

1.70 

2.10 

3.00 

Q1 

2.05 

3.05 

3.10 

4.00 

Q2  (Median) 

2.60 

3.60 

3.20 

4.35 

Q3 

3.00 

4.25 

4.00 

5.05 

HI  Score 

3.80 

10.00 

10.70 

9.70 

HI-LO 

2.40 

8.30 

8.60 

6.70 

Q3-Q1 

0.95 

1.20 

0.90 

1.05 

N 

36 

36 

36 

36 

Group 

C 

B 

A 

D 

Note .  Kruskal-Wallis  k-sample  test  of  distribution  differences  yields  H, 
which  is  distributed  as  Chi-square  with  k-1  degrees  of  freedom. 

H  *  69.68.  Thus,  p  <  .001,  since  p  ■  .001  for  H  >  16.27. 


Table  B-32 


Test  Item  Distribution  Statistics 
Task  3  (Tell  Nomenclature  of  Marked  Part) 
EXPLODED  VIEW:  SEQ/C/NE 


Statistic 

Number 

of  Callouts 

10 

27 

44 

62 

LO  Score 

1.60 

1.80 

2.00 

2.20 

Q1 

2.25 

2.15 

2.80 

3.55 

.i 

Q2  (Median) 

2.95 

2.45 

3.20 

3.90 

Q3 

3.60 

3.00 

3.60 

4.30 

HI  Score 

5.00 

3.80 

4.10 

6.00 

HI-LO 

3.40 

2.00 

2.10 

3.80 

Q3-Q1 

1.35 

0.85 

0.80 

0.75 

N 

36 

36 

36 

36 

Group 

D 

C 

B 

A 

Note.  Kruskal-Wallis  k-sample  test  of  distribution  differences  yields  H, 
which  is  distributed  as  Chi-square  with  k-1  degrees  of  freedom. 

H  *  49.59.  Thus,  p  <  .001,  since  p  *  .001  for  H  >  16.27. 


B-32 


Table  B-34 


\ 


Test  Item  Distribution  Statistics 
Task  3  (Tell  Nomenclature  of  Marked  Part) 
EXPLODED  VIEW:  SEQ/NC/NE 


Statistic 

Number 

of  Callouts 

• 

10 

27 

44 

62 

LO  Score 

1.50 

2.30 

1.80 

1.80 

Q1 

2.15 

3.20 

3.10 

2.60 

Q2  (Median) 

2.85 

3.85 

3.60 

3.00 

Q3 

3.20 

4.90 

4.10 

3.45 

HI  Score 

A. 80 

7.90 

5.50 

7.10 

HI-LO 

3.30 

5.60 

3.70 

5.30 

Q3-Q1 

1.05 

1.70 

1.00 

0.85 

N 

36 

36 

36 

36 

Group 

B 

A 

D 

C 

Note.  Kruskal-Wallis  k-sample  test  of  distribution  differences  yields  H, 
which  ij  distributed  as  Chi-square  with  k-1  degrees  of  freedom. 

H  *  35.31.  Thus,  p  <  .001,  since  p  -  .001  for  H  >  16.27. 


Table  B-35 


Test  Item  Distribution  Statistics 
Task  3  (Tell  Nomenclature  of  Marked  Part) 
EXPLODED  VIEW:  RAN/C/E 


Statistic 

Number 

of  Callouts 

10 

27 

44 

62 

LO  Score 

1.30 

2.50 

2.20 

2.40 

Q1 

1.85 

3.20 

3.10 

3.45 

Q2  (Median) 

2.20 

3.85 

3.65 

4.05 

Q3 

2.80 

4.00 

4.15 

4.55 

HI  Score 

3.20 

5.80 

5.60 

11.50 

HI-LO 

1.90 

3.30 

3.40 

9.10 

Q3-Q1 

0.95 

0.80 

1.05 

1.10 

N 

36 

36 

36 

36 

Group 

B 

A 

D 

C 

Note .  Kruskal-Wallis  k-sample  test  of  distribution  differences  yields  H, 
which  is  distributed  as  Chi-square  with  k-1  degrees  of  freedom. 

H  =  68.95.  Thus,  p  <  .001,  since  p  ■  .001  for  H  >  16.27. 


Table  B-36 


Test  Item  Distribution  Statistics 
Task  3  (Tell  Nomenclature  of  Marked  Part) 
EXPLODED  VIEW:  RAN/C/NE 


Statistic 

Number 

of  Callouts 

10 

27 

44 

62 

LO  Score 

1.70 

2.00 

1.80 

1.70 

Q1 

2.00 

2.50 

3.00 

3.05 

Q2  (Median) 

2.45 

3.00 

3.75 

3.60 

Q3 

3.00 

3.20 

4.10 

4.25 

HI  Score 

5.00 

10.10 

5.30 

7.20 

HI-LO 

3.30 

8.10 

3.50 

5.50 

Q3-Q1 

1.00 

0.70 

1.10 

1.20 

N 

36 

36 

36 

36 

Group 

A 

D 

C 

B 

Note.  Rruskal-Hallis  k-s ample  test  of  distribution  differences  yields  H, 
which  is  distributed  as  Chi-square  with  k-1  degrees  of  freedom. 


Table  B-37 


Test  Item  Distribution  Statistics 
Task  3  (Tell  Nomenclature  of  Harked  Part) 
EXPLODED  VIEW:  RAN/NC/E 


Statistic 

Number 

of  Callouts 

10 

27 

44 

62 

LO  Score 

1.30 

1.30 

2.30 

2.50 

Q1 

1.80 

2.60 

2.55 

3.60 

Q2  (Median) 

2.15 

3.00 

3.65 

4.00 

Q3 

2.80 

3.60 

4.00 

4.55 

HI  Score 

4.00 

4.90 

5.60 

6.00 

HI-LO 

2.70 

3.60 

3.30 

3.50 

Q3-Q1 

1.00 

1.00 

1.45 

0.95 

N 

36 

36 

36 

36 

Group 

D 

C 

B 

A 

Note.  Kruskal-Wallis  k-s ample  test  of  distribution  differences  yields  H 
which  is  distributed  as  Chi-square  with  k-1  degrees  of  freedom. 

H  ■  63.6.  Thus,  p  <  .001,  since  p  •  .001  for  H  >  16.27. 


B-37 


Table  B-38 


TesC  Item  Distribution  Statistics 
Task  3  (Tell  Nomenclature  of  Marked  Part) 
EXPLODED  VIEW:  RAN/NC/NE 


Statistic 

Number 

of  Callouts 

10 

27 

44 

62 

LO  Score 

1.80 

1.20 

1.80 

2.20 

Q1 

2.10 

2.00 

2.20 

2.55 

Q2  (Median) 

2.85 

2.50 

2.90 

3.10 

Q3 

3.55 

3.15 

3.45 

3.90 

HI  Score 

7.00 

4.60 

6.40 

13.70 

HI-LO 

5.20 

.  3.40 

4.60 

11.50 

Q3-Q1 

1.45 

1.15 

1.25 

1.35 

N 

36 

36 

36 

36 

Group 

C 

B 

A 

D 

Note.  Kruskal-Wallis  k-sample  test  of  distribution  differences  yields  H, 
which  is  distributed  as  Chi-square  with  k-1  degrees  of  freedom. 

H  ■  12.47.  Thus,  p  <  .01,  since  p  ■  .01  for  H  >  11.34. 


Table  B-39 


X 


Test  Item  Distribution  Statistics 
Task  4  (Use  Zone  System  to  Point  to  Part  Given  Callout  Number) 
Exploded  View  with  62  Callouts 


Statistic 

RAN/C/E 

RAN/C/NE 

RAN/NC/E 

RAN/NC/NE 

L0  Score 

5.10 

5.50 

2.70 

6.50 

Q1 

9.45 

9.05 

9.35 

8.40 

Q2  (Median) 

11.90 

12.15 

12.60 

11.16 

Q3 

18.90 

16.50 

16.60 

12.50 

HI  Score 

39.20 

23.80 

44.10 

37.60 

HI-LO 

34.10 

18.30 

41.40 

31.10 

Q3-Q1 

9.45 

7.45 

7.25 

4.10 

N 

36 

36 

36 

36 

Group 

A 

B 

C 

D 

Note.  Kruskal-Wallis  k-saaple  test  of  distribution  differences  yields  H, 
which  is  distributed  as  Chi-square  with  k-1  degrees  of  freedom. 

H  -  5.07.  Thus,  p  >  .05,  since  p  -  .05  for  H  >  7.82. 


B-39 


Table  B-40 


Test  Item  Distribution  Statistics 
Task  5  (Use  Zone  System  to  Point  to  Part  Given  Nomenclature) 
Exploded  View  with  62  Callouts 


Statistic 


RAN/C/E 


RAN/C/NE 


RAN/NC/E 


RAN/NC/NE 


Kruskal-Wallis  k-sample  test  of  distribution  differences  yields  H 
which  is  distributed  as  Chi-square  with  k-1  degrees  ofvfreedom. 

H  -  25.06.  Thus,  p  <  .001,  since  p  ■  .001  for  H  >  16.27. 


